Job Overview
We are in search of a dedicated Senior Data Engineer who thrives on innovation and is keen to tackle challenges that extend beyond conventional data engineering. Your extensive experience in constructing large-scale data pipelines, optimizing SQL, and managing cloud infrastructure will be invaluable in driving our AI capabilities. In this pivotal role, you will contribute to the development of high-performance data systems and play a crucial part in integrating advanced AI functionalities into the workflows of global data teams.
Key Responsibilities
- Design and establish scalable data pipelines capable of processing over 100,000 jobs while managing petabyte-scale data daily.
- Develop and implement a cloud-native data infrastructure on AWS, leveraging tools such as Kubernetes and Airflow for streamlined resource management.
- Create a robust SQL intelligence ecosystem that includes query optimization, dynamic pipeline generation, and data lineage tracking.
- Drive the integration of traditional data engineering practices with cutting-edge AI technologies, ensuring a forward-thinking approach to our data workflows.
Required Skills
- Over 8 years of experience in data engineering with a strong focus on building scalable data systems.
- Proficiency in Python and SQL is essential.
- Extensive background in SQL query profiling, optimization, and performance tuning, especially with Snowflake.
- In-depth knowledge of SQL Abstract Syntax Tree (AST) and experience with SQL parsers (e.g., sqlglot) for generating detailed data lineage and creating dynamic ETLs.
- Experience in developing data pipelines utilizing Airflow or dbt.
- [Optional] Familiarity with cloud platforms, primarily AWS.
- [Optional] Understanding of Kubernetes (K8s) for managing containerized deployments.
Qualifications
- Minimum of 8 years of relevant experience in data engineering.
- Solid track record in constructing scalable data pipelines and optimizing systems for performance.
- A Bachelor’s degree in Computer Science, Engineering, or a related field is preferred.
Career Growth Opportunities
This position serves as a gateway into the rapidly advancing field of AI-enhanced data engineering, providing exposure to leading-edge technologies and generative AI applications. The role offers substantial opportunities for professional development and continuous learning.
Company Culture And Values
Join a vibrant and innovative workplace where collaboration and creativity are at the forefront. Our team of skilled engineers is committed to pushing the boundaries of data engineering and fostering a culture that encourages the sharing of ideas.
Compensation And Benefits
- Competitive salary ranging from \(60,000 to \)120,000, influenced by your experience, location, and interview performance.
- Equity options between 0.0% and 0.4%.
- Opportunities to influence company direction and contribute to open-source initiatives, enhancing your professional visibility within the tech community.
How To Apply
If you believe you are the ideal candidate for this exciting opportunity, we look forward to receiving your application. Join us in creating something extraordinary!
Employment Type: Full-Time